1. Technical Field
This invention relates to a method and apparatus for representing a virtual object in a real environment, in particular for merging the virtual object with an image of the real environment generated by a recording device.
2. Background Information
Such a method and apparatus are known in the prior art in particular in connection with so-called augmented reality systems. The latter permit the merging of computer-generated, virtual information with visual impressions or an image of the real environment. The insertion of the virtual information or objects can be effected in contextual fashion and in correct perspective, i.e. adapted and derived from the real environment being viewed. The virtual information employed can be basically any type of data such as objects, texts, images, etc. The real environment can be captured for example with a camera worn by the user. It is further known in particular to superimpose virtual models with reality for the purpose of checking the correctness of models or of reality produced according to a specification. Ranges of application of this technology are for example factory planning (DE 101 28 015 A1, DE 10 2004 046 144 A1) or automobile design (DE 202 03 367).
Furthermore, there are known from the prior art augmented reality systems for product presentation, as described e.g. in DE 103 01 849 B4. Herein there is described a method for representing a furnishing in a usage environment wherein a reference geometry is employed in the usage environment, the usage environment including the reference geometry is optically recorded by means of a stationary image recording device and the recording is stored in the form of recording image data, the position of the stationary image recording device with regard to the reference geometry is ascertained by image processing, and the image data of the virtually existing furnishing and the recording image data are joined together in correct perspective. The usage environment is then displayed to the user with the furnishing inserted in correct perspective. Advantages of the described system and method consist in particular in the joint perception of products in the usage environment. This leads to an intuitive assessment of the suitability of various products for a purpose of use. Furthermore, it is very easy to ascertain the geometrical suitability of a product at a place of use. Further, it is possible to determine the suitability of a product for a purpose of use, e.g. the suitability of a furniture piece, without previous delivery. Any collisions of the products with other objects can be quickly discovered by the user.
With regard to any collisions of the products with other objects there is often the problem that concealments cannot be represented in a way close to reality or faithful to reality. For example, in a collision case a virtual object cannot be concealed by a real object when this is required by a perspectively correct representation, but the virtual object would always be represented in the foreground and thus cover the real object. This leads to a rather unrealistic representation. Likewise, it can be desirable to remove certain real objects (“spurious objects”), e.g. when an existing furnishing is to be replaced by a new furnishing.
From DE 10 2005 011 126 A1 there is known a method which contains a model of reality wherein mutual concealments of the real environment and of the virtual object are provided in dependence on a position of the virtual object with regard to the real environment. The research project ARIS has also dealt with augmented reality. The topic of concealment (known as “occlusion”) was already addressed therein: The user “scans” his environment. There is effected a consideration of the lighting conditions and a reproduction of the room (not described how) and an application for occlusion and object collision. A camera calibration is effected via vanishing points and the user is able to remove objects from the image of his environment.
Lepetit, V., Berger, M.-O., Handling occlusion in augmented reality systems: a semi-automatic method. In Augmented Reality, 2000. (ISAR 2000). Proceedings. IEEE and ACM International Symposium on Augmented Reality. Publication date: 2000, pages 137-146, Meeting date: Oct. 5, 2000-Oct. 6, 2000, ISBN: 0-7695-0846-4: In this publication the occluding object must be initialized by hand. Automatic acceptance of the information is only possible for small changes in the camera pose. Initialization by hand is also required by the following method: Resolving occlusion in image sequence made easy: Kiem Ching Ong, Hung Chum Teh, Tiow Seng Tan. In: The Visual Computer (1998) 14:153±165, Springer-Verlag 1998.
Handling Occlusions in Real-time Augmented Reality: Dealing with Movable Real and Virtual Objects, Pierre-Alexandre Fortin and Patrick Hebert. In Proceedings of the 3rd Canadian Conference on Computer and Robot Vision (CRV'06) 0-7695-2542-3/06 ©2006 IEEE. In this publication the occlusion problem is solved within very controlled and restricted basic conditions. Two approaches are compared: One approach works with a 3D initialization geometry which is tied to optical markings with which the spurious object must be labeled. The initialization geometries are then combined with a background subtraction algorithm. However, the background must be uniform or the visualization camera must not move. For automatic, realistic representation of any objects, this method is unsuitable. The second approach uses depth information from a stereo camera system and employs it directly for occlusion. However, due to noise and the well-known problems of using depth cameras, this approach is qualitatively dissatisfactory.
With regard to the removal of spurious objects it is likewise known in the prior art to use segments. The main focus of the following publication is an approach for generating texture information. However, the identification of the replaced segment is not generally applicable to spurious objects, but rather only the marking is removed therein which is easy to recognize for image processing algorithms and is employed for generating the camera pose. (Siltanen, Sanni: “Texture Generation over the Marker Area” in IEEE/ACM International Symposium on Mixed and Augmented Reality, 2006. ISMAR 2006. Publication date: October 2006, pages: 253-254, Place: Santa Barbara, Calif., USA, ISBN: 1-4244-0651-X)
An object of the invention is to specify a method and apparatus for representing a virtual object in a real environment with which any occurring collisions of virtual objects with real objects upon merging with a real environment can be represented in a way largely close to reality.